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DETAILED ACTION 



Information Disclosure Statement 

I. The information disclosure statement (IDS) submitted on 3/24/03 and 11/7/03 are in 
compliance with the provisions of 37 CFR1.97. Accordingly, the information disclosure 
statements have been considered by the examiner. 

Specification 

II. The title of the invention is not descriptive. A new title is required that is clearly 
indicative of the invention to which the claims are directed. It is suggested Applicant insert 
language which is directed to the affecting of multimedia data in response to user physiological 
data. 

Claim Rejections - 35 USC §102 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

III. Claims 1-5, 11-13, 15, 25-29, 34, and 36 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Zawilinski. (US Pat. No. 5,676,138). 
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As to Claim 1 5 Zawilinski discloses a multimedia system comprising an emotional 
response analyzer. The system further comprises sensors for sensing the user's physical 
reactions (i.e., gaze, galvanic skin response, etc.) to various stimuli, including commercials, 
wherein said stimuli are associated with recognizable responses (i.e., surprise, anger, disgust, 
etc.). Moreover, this information can be indexed and associated with whatever stimuli were 
being displayed at the time of said response. (Fig. 1 & 2; Abstract; Col. 1, Ln. 6-47; Col. 3, Ln. 
39-Col. 4, Ln. 47; Col. 5, Ln. 34-43, 63-Col. 6, Ln. 26, 35-Col. 7, Ln. 5; Col. 9, Ln. 50-64). 
Accordingly, Zawilinski anticipates each and every limitation of Claim 1. 

Claims 11, 12, 26, and 34 are encompassed within the limitations of Claim 1. Thus, each 
is analyzed and rejected as previously discussed. 

The limitations of Claim 2 are inherent in view of the rejection of Claim 1. In order for 
the system to be capable of indexing and associating user responses with various stimuli, it must 
have some form of memory. Accordingly, Zawilinski anticipates each and every limitation of 
Claim 2. 

As to Claim 3, Zawilinski further discloses the system comprises a plurality of sensors. 
(Col. 6, Ln. 3-26). Accordingly, Zawilinski anticipates each and every limitation of Claim 3. 

The limitations of Claim 4 are inherent in view of Claim 3. If the system has 
physiological sensors, it must have some form of receiver for receiving said signals. 
Accordingly, Zawilinski anticipates each and every limitation of Claim 4. 

As to Claim 5, Zawilinski further teaches the system can analyze various physiological 
data including heart rate, galvanic skin response, etc. (i.e., aggregation of signals), (citations of 
Claim 1). Accordingly, Zawilinski anticipates each and every limitation of Claim 5. 
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As to Claim 13, Zawilinski further teaches the system can plot various stimuli changes 
over time (i.e., changes in condition relative to a baseline level). (Col. 9, Ln. 12-20). 
Accordingly, Zawilinski anticipates each and every limitation of Claim 13. 

As to Claim 15, Zawilinski further teaches the system can analyzed the heart rate of a 
user, (citations of Claim 1). Accordingly, Zawilinski anticipates each and every limitation of 
Claim 15. 

As to Claim 26, Zawilinski further teaches the system can analyzed the galvanic skin 
response of a user. (Col. 6, Ln. 10-19). Accordingly, Zawilinski anticipates each and every 
limitation of Claim 26. 

As to Claims 27 and 28, Zawilinski further teaches the system is capable of measuring 
the gaze of the viewer and the electrical activity in muscles, which relates to negative/positive 
facial expressions. (Col. 4, Ln. 13-31 and citations of Claim 1). Thus, in essence, the system 
monitors gazing and facial expressions, which are both observable responses. Accordingly, 
Zawilinski anticipates each and every limitation of Claims 27 and 28. 

As to Claim 29, Zawilinski further teaches the system can track the direction of the user's 
gaze. (Col. 9, Ln. 50-64). Accordingly, Zawilinski anticipates each and every limitation of 
Claim 29. 

As to Claim 36, Zawilinski further teaches the system can analyze the electrical activity 
of a user muscles, including those used when laughing (i.e., which is an audibly observable 
response). (Col. 4, Ln. 14-31). Accordingly, Zawilinski anticipates each and every limitation of 
Claim 36. 
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Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 

This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (£) or (g) prior art under 35 U.S.C. 103(a). 

IV. Claims 6-10, 16, 17, 30, and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Zawilinski in view of Kazama et al. (US Pat. No. 6,1 1 1,580). 
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Claim 6 recites the system of Claim 1, wherein at least on sensor is a video image 
capturing device. As discussed above, Zawilinski anticipates each and every limitation of Claim 
1, but fails to specifically recite the limitations of Claim 6. However, within the same field of 
endeavor, Kazama et al disclose a similar system, which comprises a video camera. (Col. 3, Ln. 
34-37). Accordingly, it would have been obvious to one having ordinary skill in this art at the 
time of Applicant's invention to combine the systems of Zawilinski and Kazama in order to 
provide a system which allows for a visual analysis of a viewer's responses to programming 
data. 

Claim 16 corresponds to Claim 6. Thus, it is analyzed and rejected as previously 
discussed. 

As to Claim 7, Kazama further discloses the system analyzes images of a user's face to 
determine gaze direction and gesturing (i.e., library of images used for comparisons). (Col. 3, 
Ln. 38-Col. 4, Ln. 46; Col. 5, Ln. 26-33). Accordingly, the combined systems of Zawilinski and 
Kazama disclose all limitations of Claim 7. 

Claim 17 corresponds to Claim 7. Thus, it is analyzed and rejected as previously 
discussed. 

As to Claim 8, it is inherent the system of Kazama comprise a recorder for recording the 
images in order to determine the gaze and gesturing of the user, (citations of Claim 7). 
Accordingly, the combined systems of Zawilinski and Kazama disclose all limitations of Claim 
8. 

As to Claim 9, Kazama further teaches the system is capable of identifying speech 
signals. Thus, it is inherent the system comprise some form of microphone for picking up said 
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speech signals. (Abstract; Col. 6, Ln. 32-44). Accordingly, the combined systems of Zawilinski 
and Kazama disclose all limitations of Claim 9. 

Claim 35 corresponds to Claim 9. Thus, it is analyzed and rejected as previously 
discussed. 

As to Claim 10, Kazama further discloses the system comprises a pressure sensor (i.e., 
senses pressure of viewing environment) used to analyzed the viewer's attention status. (Col. 10, 
Ln. 40-65). Accordingly, the combined systems of Zawilinski and Kazama disclose all 
limitations of Claim 10. 

As to Claim 30, Kazama further discloses the system is capable of analyzing the gaze of 
the viewer in order to ascertain the attention period (i.e., duration of viewer's directional gaze 
before changing to different direction). (Col. 3, Ln. 39-Col. 4, Ln. 7). Accordingly, the 
combined systems of Zawilinski and Kazama disclose all limitations of Claim 30. 

V. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over Zawilinski in 
view of Ark et al. (US Pat. No. 6,190,314). 

Claim 14 recites the method of Claim 11, wherein the physical condition is body 
temperature. As discussed above, Zawilinski anticipates each and every limitation of Claim 1, 
but fails to disclose the limitation of Claim 14. However, within the same field of endeavor, Ark 
et al disclose a similar system which analyzes the user's temperature. (Abstract; Col. 2, Ln. 39- 
41; Col. 4, Ln. 27-34). Accordingly, it would have been obvious to one having ordinary skill in 
this art at the time of Applicant's invention to combine the systems of Zawilinski and Art in 
order to provide a system which can analyze a variety of physiological data. 
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VI. Claims 18-24 are rejected under 35 U.S.C. 103(a) as being unpatentable over Zawilinski 
in view of Abecassis. (US Pat. No. 5,664,046). 

Claim 18 recites the method of Claim 11, further comprising limitations which will not 
be recited here, however each will be addressed in turn. As discussed above, Zawilinski 
anticipates each and every limitation of Claim 1, and further teaches said physiological responses 
can be associated with a preference level (i.e., various values given to responses). (Fig. 5). But, 
Zawilinski fails to disclose the remaining limitations of Claim 18. However, within the same 
field of endeavor, Abecassis discloses a similar system wherein the multimedia data is 
segmented according to a viewer's content preferences. Subsequently, the system analyzes this 
data to provide alternative program segments which meet said viewer preferences (i.e., enhances 
program selection). (Abstract; Col. 5, Ln. 40-67). Accordingly, it would have been obvious to 
one having ordinary skill in this art at the time of Applicant's invention to combine the systems 
of Zawilinski and Abecassis in order to provide a system which can provide alternate 
programming based upon a viewer's physiological responses to programming. 

As to Claim 19, Abecassis further teaches that segments containing objectionable rating 
levels (i.e., distinguishing characteristics) are flagged (i.e., notification) in advance, thus 
allowing the editor to review them. (Col. 23, Ln. 54-Col. 24, Ln 44). Accordingly, the 
combined systems of Zawilinski and Abecasssis disclose all limitations of Claim 19. 

Claim 24 corresponds to Claim 19. Thus, it is analyzed and rejected as previously 
discussed. 



NFR-PG.8of 12. 



Application/Control Number: 10/014,179 Page 9 

Art Unit: 2617 

As to Claim 20, Abecassis further teaches that, upon detecting an objectionable segment, 
the system will insert an acceptable segment, (citations used under Claim 18). Accordingly, the 
combined systems of Zawilinski and Abecasssis disclose all limitations of Claim 20. 

As to Claim 21, the Examiner takes Official Notice that, at the time of Applicant's 
invention, it was notoriously well known in this art to transmit program rating information (i.e., 
characteristic info) along with EPG data. Accordingly, it would have been obvious to one 
having ordinary skill in this art at the time of Applicant's invention to modify the combined 
systems of Zawilinski and Abecassis, thereby providing a system which allows the user to view 
rating information via the EPG. 

As to Claim 22, Abecassis further teaches that segment data can be derived from 
programming video or audio. (Col. 7, Ln. 42-51). Moreover, as discussed under Claim 21, 
rating info can be derived from EPG (i.e., textual) data. Accordingly, the combined systems of 
Zawilinski and Abecasssis disclose all limitations of Claim 22. 

As to Claim 23, Abecassis further discloses the system comprising a storage unit which 
automatically records programming segments. (Abstract; Col. 14, Ln. 12-42). The remainder of 
the limitations are encompassed within the limitations of Claim 18. Accordingly, the combined 
systems of Zawilinski and Abecasssis disclose all limitations of Claim 23. 

VII. Claims 31 and 32 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Zawilinski in view of Black et al. (US Pat. No. 5,774,591). 

Claim 31 recites the method of Claim 27, wherein the visually observable response 
includes the furrowing of the viewer's brow. As discussed above, Zawilinski anticipates each 
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and every limitation of Claim 27, and further teaches the system can analyzed the electrical 
impulses in the user's muscles (i.e., which can related to facial expressions), but fails to 
specifically recite analyzing eyebrow furrowing. However, within the same field of endeavor, 
Black et al disclose a similar system which analyzes changes in the viewer's eye brows. 
(Abstract; Col. 7, Ln. 1-45; Col. 26, Ln. 50-67; Col. 28, Ln. 28-62). Accordingly, it would have 
been obvious to one having ordinary skill in this art at the time of Applicant's invention to 
combine the systems of Zawilinski and Black in order to provide a system capable of analyzing 
facial features, thereby affecting the operation of a computer system. 

The limitations of Claim 32 would be obvious variants of the limitations of Claim 31. 
Since Black et al allow for the analysis of user eyebrows, analyzing the depth of movement in 
said eyebrows would only be an obvious variant. Accordingly, it would have been obvious to 
one having ordinary skill in this art at the time of Applicant's invention to modify the combined 
systems of Zawilinski and Black in order to provide a system which analyzes the depth of 
furrows, thus allowing for an analysis of the level of a viewer facial response (i.e., level of anger, 
surprise, understanding, etc). 

VIII. Claims 33 and 38 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Zawilinski in view of Bentolila et al. (US Pat. Pub. No. 2003/0101449). 

Claim 33 recites the method of Claim 11, wherein the associating step is performed at 
least in part by using a Hidden Markov Model technique. As discussed above, Zawilinski 
anticipates each and every limitation of Claim 1 1, but fails to discuss the limitations of Claim 33. 
However, within the same field of endeavor, Bentolila et al disclose a similar system which 
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utilizes a Hidden Markov technique. (Par. [0021]). Accordingly, it would have been obvious to 
one having ordinary skill in this art at the time of Applicant's invention to combine the systems 
of Zawilinski and Bentolila in order to provide a system for user profile data prediction. 

Claim 38 corresponds to Claim 33. Thus, it is analyzed and rejected as previously 
discussed. 

IX. Claim 37 is rejected under 35 U.S.C. 103(a) as being unpatentable over Zawilinski in 
view of Shinohara. (US Pat. Pub. No. 2003/0005431). 

Claim 37 recite the method of Claim 34, wherein the audibly observable response is the 
inflection (i.e., changes in pitch or tone) of a listener's vocalization, tending to indicate a 
question has been enunciated. As discussed above, Zawilinski anticipates each and every 
limitation of Claim 34, but fails to specifically recite the limitations of Claim 37. However, 
within the same field of endeavor, Shinohara discloses a similar system which analyzes spectral 
data related to speech patterns, such as pitch, tone, frequency, etc., in order to identify a 
television viewer. (Par. [0026]). Accordingly, it would have been obvious to one having 
ordinary skill in this art at the time of Applicant's invention to combine the systems of 
Zawilinski and Shinohara in order to provide a system which is capable of analyzing various 
tones, pitches, etc. of a given voice. 

Conclusion 

The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 
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a. Agnihotri et al (US Pat. Pub. No. 2002/0178440) disclose a similar system. 

b. Orbach et al (US Pat. No. 5,896,164) disclose a video system which provides 
biofeedback. 

c. Claessens (US Pat. No. 6,516,464) discloses a system for detecting audience 
response to various stimuli. 

d. Schmidt et al (US Pat. No. 5,243,517) disclose a system for evaluating 
physiological data in response to stimulus provided by multimedia data. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jade O. Laye whose telephone number is (571) 272-7303. The 
examiner can normally be reached on Mon. 7:30am-4, Tues. 7:30-2, W-Fri. 7:30-4. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Chris Kelley can be reached on (571) 272-7331. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 

Application Information Retrieval (PAIR) system. Status information for published applications 

may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 

applications is available through Private PAIR only. For more information about the PAIR 

system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 

system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

Examiner: Jade O. Lave 
December 21, 2005. 

VIVEK SRIVASTAVA 

PRIMARY EXAMINER 
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